Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added how to map AniDB specials other than S specials #287

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

sixtenbe
Copy link
Collaborator

Got it from a comment ScudLee placed on Issue #150. Leaving this as a Pull request since I'd like verification that this is 100% correct and others feel it's placed appropriately in the readme.

sixtenbe added 2 commits July 14, 2019 00:37
Added how to map anidb non S-specials as I've understood it.
@purposelycryptic
Copy link
Contributor

Quick question - since the mapping schema from the issue mentioned is based on HAMA (Or rather ASS, its Episode Scanner), would it be possible to expand this to fully align with it?

The full ASS-HAMA mapping schema is as follows:

Type Internal letter Episode number
Specials S Episodes 001-100
OPs C Episodes 101-150
EDs C Episodes 151-200
Trailers T Episodes 201-300
OPs/EDs P Episodes 301-400
Others O Episodes 401-500
unmapped Episodes 501-600

Obviously, 'Unmapped' specials would be the exception, as those are already being mapped to either S00E00 or S00E99, and, with that being the case, the range of the regular 'Specials' specials would need to be restricted to S00E01-S00E98, so the schema would look something like this then:

Type Internal letter Episode number
Specials S Episodes 001-098
OPs C Episodes 101-150
EDs C Episodes 151-200
Trailers T Episodes 201-300
OPs/EDs P Episodes 301-400
Others O Episodes 401-500
unmapped Episode 000 or 099

I've done a cursory check, and it seems this schema has already been at least partially used for the list, with 182 instances of C-specials (OPs and EDs) using 1xx episode numbers across 10 series, 16 instances of T-specials (Trailers and PVs) using 2XX episode numbers across 7 series, and 8 instances of O-Specials using 4XX episode numbers across 2 series (Which can be anything - currently on the list it is 7 instances of Music Videos from Furi Kuri, and one incorrect use for a Trailer (AniDB, TVDB) from The Animatrix).

There are actually quite a few O-specials on AniDB which can be mapped to TheTVDB, though - they are frequently used when something is released in different forms, such as with The Animatrix, which, in addition to 9 individual episodes, was also released as 2 55-minute episodes, which are listed on AniDB as O1 and O2, respectively. For a more recent example, Owarimonogatari's episodes, on both AniDB and TheTVDB, are each 25 minutes long, with the exception of the first, which was 50 minutes long. But for the BD release, it was then seemingly split into 2 episodes, which now exist on AniDB as O1 and O2. Having concrete info for mapping these in the readme should hopefully lead to more of these types of episodes being mapped.

The biggest question then is in regard to C-specials, which currently aren't divided between OPs and EDs - it would not take much effort to edit the small number of current EDs included in the Lists so they would also comply with the numbering scheme, and it would certainly make the mappings easier to use and fix when necessary, given how often OPs and EDs tend to be renumbered on AniDB. I'd be happy to do so, if agreement can be reached on the matter.

@sixtenbe
Copy link
Collaborator Author

HAMA and its Scanner ASS are not a authoritative source on how to map AniDB specials. An authoritative source would be ScudLee or the Kodi extension he wrote, which this mapping list was made for.

There is no reason to believe that Endings episodes should be mapped as 151+ and if you look on AniDB they are all internally C1+ with a title differentiating if they're an opening or ending.

e.g. Take Angel beats ED1a If you look on the page anime page it's episode ED1a, but behind that it's actually listed as C3 with the title "Ending 1a", with the title displayed on the anime page being the song.

For The Animatrix I don't see what you mean by incorrect use of trailer mapping and the O1 O2 mappings are not something obvious, since it's a one-to-many mapping not many-to-one or one-to-one. I don't expect clients to be able to handle a mapping like:
;401-1;401-2;401-3;401-4;401-5;402-6;402-7;402-8;402-9;
I'd expect many to take just use the two last mapping for the episodes 401-5 and 402-9 and even if you have some handling, do you want to spam in data from 5 episodes into 1 entry. There is no obvious best choice considering how the data needs to be processed by other apps.

Actually looking a bit more it might be better to just copy from one of the readme on ScudLee-XBMC-Addons placing the following table in the same in the mapping-list section

OPs/EDs, trailers, and more can be treated as special episodes. Just start numbering them from 100+.

Type Episode number
OPs/EDs "C" Season 0 Episodes 101-199
Trailers "T" Season 0 Episodes 201-299
Parodies "P" Season 0 Episodes 301-399
Other "O" Season 0 Episodes 401-499

@purposelycryptic
Copy link
Contributor

HAMA and its Scanner ASS are not a authoritative source on how to map AniDB specials. An authoritative source would be ScudLee or the Kodi extension he wrote, which this mapping list was made for.

There isn't really an "authoritative source" on how to map AniDB specials, since it is entirely dependent on how the client program/plug-in/scanner interprets the data - ScudLee has not been active on GitHub this year, and the Kodi scraper has not been updated since 2016, has not worked properly in over 2 years, and attempted usage quickly results in being http-banned from AniDB.

It has been long dropped from Kodi's pinned scrapers, and has been considered abandoned by the Kodi Team for quite some time now (One of their members mentioned not seeing or hearing from anyone working on the plugin for months, and that was back in February - no change since). The last post in the thread for the scraper was in April, and that was another unanswered cry for help. The last topic mentioning the scraper is from May of last year. Its entry has been removed from the Kodi wiki entirely, even from the 'Broken Add-Ons' category.

The main integrated anime plugin in use for Kodi seems to be Nakamori, which is directly linked to the user's Shoko Server instance, and gets its metadata from there.

AFAIK, the primary systems that currently actually use the anime_lists are HAMA, and things like Sonarr, which use it in their optional plug-ins?

There is no reason to believe that Endings episodes should be mapped as 151+ and if you look on AniDB they are all internally C1+ with a title differentiating if they're an opening or ending.

e.g. Take Angel beats ED1a If you look on the page anime page it's episode ED1a, but behind that it's actually listed as C3 with the title "Ending 1a", with the title displayed on the anime page being the song.

That is exactly the reason for mapping EDs to a separate range from OPs, and exactly why ASS uses that scheme - I've been using/contributing to AniDB for a fair amount of time (going on ten years now), and the C-episodes have always presented a headache for everyone, from contributors to mods; it's part of the basic DB scheme, so it can't realistically be changed (Not even with a massive effort like back when Categories and Tags were merged into a single system back in 2014, which resulted in issues that still haven't been entirely cleaned up), but it was implemented that way back when separate OP/ED files were all but non-existent, and no one thought there would eventually be so many that they might each need their own category.

The problem with using a single range for both is, AniDB entries' C-range changes, receiving additions as well as episode merges - and a lot more often than you might expect, as well, even for series over half a decade old. That wouldn't be much of an issue if they had independent categories, as adding a hypothetical OP4, "Opening 2b" would simply result in going from

ID Episode
OP1 Opening 1
OP2 Opening 2
OP3 Opening 3

to

ID Episode
OP1 Opening 1
OP2 Opening 2a
OP3 Opening 3
OP4 Opening 2b

which would either stay that way, or receive an episode edit, becoming:

ID Episode
OP1 Opening 1
OP2 Opening 2a
OP3 Opening 2b
OP4 Opening 3

But adding a C-episode is different, as when you add the same episode to the C-range:

ID Episode
C1 Opening 1
C2 Opening 2
C3 Opening 3
C4 Ending 1
C3 Ending 2
C4 Ending 3

you get

ID Episode
C1 Opening 1
C2 Opening 2a
C3 Opening 3
C4 Ending 1
C3 Ending 2
C4 Ending 3
C5 Opening 2b

which, to maintain continuity, then receives an episode epit, becoming:

ID Episode
C1 Opening 1
C2 Opening 2a
C3 Opening 2b
C4 Opening 3
C3 Ending 1
C4 Ending 2
C5 Ending 3

In other words, C4 goes from being an ED to a OP. Using the combined mapping, the changeover point between the two ranges is lost, as S0E104 was not specifically defined as either an OP or ED:

Mapping ID Episode -> Mapping ID Episode
101 C1 Opening 1   101 C1 Opening 1
102 C2 Opening 2   102 C2 Opening 2a
103 C3 Opening 3   103 C3 Opening 2b
104 C4 Ending 1   104 C4 Opening 3
105 C3 Ending 2   105 C3 Ending 1
106 C4 Ending 3   106 C4 Ending 2
        107 C5 Ending 3

But, unlike AniDB, where separating the two is now impossible, it is very simple to do here:

Mapping ID Episode -> Mapping ID Episode
101 C1 Opening 1   101 C1 Opening 1
102 C2 Opening 2   102 C2 Opening 2a
103 C3 Opening 3   103 C3 Opening 2b
151 C4 Ending 1   104 C4 Opening 3
152 C3 Ending 2   151 C3 Ending 1
153 C4 Ending 3   152 C4 Ending 2
        153 C5 Ending 3

This makes it much easier for a metadata agent to identify the type of episode being referenced. The issue that AniDB has can be easily side-stepped.

For The Animatrix I don't see what you mean by incorrect use of trailer mapping[...]

For The Animatrix, the current mapping is:

  <anime anidbid="517" tvdbid="80468" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
    <name>The Animatrix</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-0;401-1;</mapping>
    </mapping-list>
  </anime>

But the AniDB data looks like this:
animatrix-anidb
And the TVDB data looks like this:
animatrix-tvdb

Currently, it is mapping O1 - "Episodes 1-9 (Part 1 of 2)" to a Trailer. That is incorrect. It should be:

  <anime anidbid="517" tvdbid="80468" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
    <name>The Animatrix</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-0;201-1;</mapping>
    </mapping-list>
  </anime>

[...] and the O1 O2 mappings are not something obvious, since it's a one-to-many mapping not many-to-one or one-to-one. I don't expect clients to be able to handle a mapping like:
;401-1;401-2;401-3;401-4;401-5;402-6;402-7;402-8;402-9;
I'd expect many to take just use the two last mapping for the episodes 401-5 and 402-9 and even if you have some handling, do you want to spam in data from 5 episodes into 1 entry. There is no obvious best choice considering how the data needs to be processed by other apps.

And yet, that is precisely what we just went with for "Saiki Kusuo no Sainan" in issue #251, with commit a13bbaa.

HAMA, at the very least, handles this kind of mapping perfectly well. For other agents, I do not know, but it is up to them as to whether they support it or not - the alternative is to have no mapping for the episodes at all, which would result in them being categorized as a generic unmapped file, without any metadata whatsoever.

Actually looking a bit more it might be better to just copy from one of the readme on ScudLee-XBMC-Addons placing the following table in the same in the mapping-list section

OPs/EDs, trailers, and more can be treated as special episodes. Just start numbering them from 100+.

Type
Episode number

OPs/EDs "C"
Season 0 Episodes 101-199

Trailers "T"
Season 0 Episodes 201-299

Parodies "P"
Season 0 Episodes 301-399

Other "O"
Season 0 Episodes 401-499

Given that is from a readme for an agent that is no longer in use, functional or supported, and has not seen any development since 2016, this seems a rather poor choice, especially since traditional parody P-episodes are no longer really a thing (just try adding one to AniDB, and se what happens), and are deleted the moment they are added to TheTVDB, as part of their core policy; that is why ASS uses the same range for OPED combined releases instead - while not incredibly common, they are still released with fair frequency (I just saw one a few days ago as part of Ohys-Raws' release of "Strike the Blood III" - Strike the Blood III - OP&ED (BD 1920x1080 x264 FLAC).mkv).

Of course, any form of OP or ED newly added to TheTVDB is also currently deleted on sight, and the individual that added it given a warning that continued attempts to do so, much like with Parody episodes, will result in account deletion; the main difference is that existing OP/ED entries are largely given a pass, whereas Parody episodes ar hunted down with extreme prejudice (They are extremely strict about those).

In the end, this is a question of whether to actively support a mapping scheme used by a dead agent that is not only no longer in use, but no longer functional, vs. that of the main agent making use of these lists today, which is under continuous development, and shows no sign of stopping in the foreseeable future.

If you're not comfortable making the call, you can always just use this:

Type Internal letter Episode number
Specials S Episodes 001-098
Trailers T Episodes 201-300
Others O Episodes 401-500
unmapped   Episode 000 or 099

It is not like OP/ED or Parody episodes are still actively being added to TheTVDB, or have been in many years - at least as far as the anime-lists go, it is strictly an issue of legacy support (and for Parody episodes, not even that). The important parts, that are still actively in use, AND will continue to be represented and mappable on both AniDB and TheTVDB, are the S-, T- and O-episodes.

@sixtenbe
Copy link
Collaborator Author

You make a good argument for changing the mapping style of C-specials, but apologies, I was added as a collaborator to help maintain the repo, not to change anything, so I don't feel fully comfortable to just change the mapping of C-specials, even if it would be a better style of mapping.

With The Animatrix the current entry is:

<anime anidbid="517" tvdbid="80468" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
  <name>The Animatrix</name>
  <mapping-list>
    <mapping anidbseason="0" tvdbseason="0">;1-0;201-1;</mapping>
  </mapping-list>
</anime>

Which would appear to be correct and I don't know how you got it to 401-1.

As for the BD episodes, I can't consider them the same as #251, as that had many AniDB entries mapped to 1 tvdb entry, which when your data presentation is based around files i.e AniDB entries, is correct even without any special handling. With the reverse, as in the case of The Animatrix, an app would have to concatenate the metadata from the tvdb entries, but I guess this should primarily be considered a problem for apps to handle and the mapping should just be the best estimate of which file covers which tvdb entries best.

@purposelycryptic
Copy link
Contributor

I went back across my various local commits, and I honestly have no idea where the 401-1 mapping for the AniMatrix came from either; I did my search for C, T, P, O episode mappings using simple regex, i.e., ;\d{3}-, then looked for individual types using ;1\d{2}- for C-Episodes, ;2\d{2}- for T-episodes, etc.

I could have sworn I found it along with the "Furi Kuri" O-episodes, and I know I copy/pasted it from somewhere, but... I can't even find it in my correction list now.

As for the the AniMatrix combined, episodes, you are completely correct - many-to-one is one thing, one-to-many is completely different. Emby used to be great at handling that sort of thing (sadly terrible with anime otherwise), in a ;401->1-4;402->6-9; sort of way, concatenating the summaries, but obviously, the anime-lists are not really structured to allow for that.

I'll take this opportunity to feel slightly embarrassed, and to remind myself that being seriously ill has a wide variety of negative effects that we don't always realize we are under the effect of at the time, including high fever; given both these initial mistakes came up the same day, I'm guessing I was more out of it than I realized.

I still think O-episode mappings for cases like Owarimonogatari (where it is a simple many-to-one situation) should be included, but the inverse is a different story; the fact that I was unable to differentiate the two troubles me, however, and, already being unable to work, and spending much of my time going from doctor to doctor, the idea that I might be in bad enough shape to not even be able to properly do something like this is somewhat terrifying.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants